The notion of relative entropy of states is a generalization of the notion of entropy to a situation where the entropy of one state is measured “relative to” another state.
It is also called
Kullback-Leibler divergence,
information divergence, or
information gain.
For two finite probability distributions and , their relative entropy is
Alternatively, for two density matrices, their relative entropy is
For a measurable space and and two probability measures on , such that is absolutely continuous with respect to , their relative entropy is the integral
where is the Radon-Nikodym derivative of with respect to .
Let be a von Neumann algebra and let , be two states on it (faithful, positive linear functionals).
The relative entropy of relative to is
where is the relative modular operator? of any cyclic and separating vector representatives and of and .
This is due to (Araki).
This definition is independent of the choice of these representatives.
In the case that is finite dimensional and and are density matrices of and , respectively, this reduces to the above definition.
The machine learning process has been characterized as a minimization of relative entropy (Ackley, Hinton and Sejnowski 1985).
Relative entropy of states on von Neumann algebras was introduced in:
A characterization of relative entropy on finite-dimensional C-star algebras is given in
A survey of entropy in operator algebras is in
A characterization of machine learning as a process minimizing relative entropy is proposed in
Last revised on December 30, 2023 at 21:36:41. See the history of this page for a list of all contributions to it.